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Dear Admiral Barnett, 

Verizon understands the critical importance of providing reliable 9-1-1 services to 9-1-1 
centers, also known as Public Safety Answering Points ("PSAPs"), and the community. Verizon 
has provided safe, reliable 9-1-1 sendee since it first became available in the 1 970s and has been a 
dedicated partner of the Commission and the public safety community in quickly responding to 
major public safety events, including 9/11, Hurricane Katrina, and the earthquake in Haiti. 
Likewise, Verizon has long been engaged with stakeholders in the public safety community, 
technology vendors, and device manufacturers to advance emergency response communications 
and to provide innovative solutions, such as the deployment of IP-based, Next Generation 9-1-1 
technologies. In light of the events of January 26, 201 1 , Verizon has redoubled its efforts to 
ensure that callers receive the level of service they deserve and expect when they call 9-1 -L 

During the heavy snowstorm in the Washington D.C. metropolitan area on January 26, an 
extremely high number of wireless 9-1-1 calls were placed to PSAPs in Montgomery and Prince 
George's counties. This spike in wireless calls to 9-1-1 caused some trunks serving those PSAPs 
that carry calls from wireless carriers to be taken out of service for varying periods of time from 
around 5:45 to 1 1 p.m. During this time, Verizon was aware of the impact to the PSAPs" 1 wireless 
trunks and focused its efforts on restoring those trunks to service. While Verizon was able to 
restore some trunks, the sustained high volume of wireless 9-1-1 calls caused many of the trunks to 
go back out of service until approximately 1 1 p.m., when the call volume diminished. 

Since that date, Verizon has been working with the Montgomery County and Prince 
George's County PSAPs to analyze what occurred on January 26 and to discuss appropriate 
measures to prevent similar events from occurring in the future. Prior to the January 26 
snowstorm, Verizon had already begun examining the relationship between a significant volume of 
wireless 9-1-1 calls, and its impact on PSAP trunks. Verizon has been working with equipment 
manufacturers to study the basis of that relationship in a lab and to devise a plan that would 
mitigate the impact on PSAPs. 
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This effort has allowed Verizon to implement its plan, which would prevent all but one of a 
PSAP's trunks from going out of service during episodes of high call volume. Verizon has 
recently tested this plan, confirmed that it is effective, and is working with various local and state 
officials to implement it, as appropriate. Verizon will keep the Commission apprised of the 
progress of those efforts. 

January 26, 2011 Focused Overload 

On January 26, 201 1 , there was a severe snowstorm in the Washington D.C. metropolitan 
area that grew in intensity during the evening rush hour. Thousands of motorists were stranded on 
area roads causing widespread traffic jams. The region also experienced extensive power outages. 
This confluence of events led to a substantial number of wireless calls being placed to 9-1-1 in 
Montgomery and Prince George's counties. For instance, the Montgomery County PSAP has 
publicly stated that it answered over three times the typical volume of calls from 8 p.m. to 8:30 
p.m. Verizon's network data not only confirms this volume, but also indicates that a significant 
number of calls were routed to busy treatment before any trunks were out of service because of the 
increased call volume. 

Each call to a PSAP is delivered over a separate trunk. The network is designed for calls to 
route to busy treatment when there are more 9-1-1 calls than available trunks serving the PSAP. In 
general, Verizon works with PSAPs to design and engineer E9-1-1 trunk groups to provide the 
National Emergency Number Association's (NENA) P. 01 Grade of Service standard, which is 
determined by the average busy hour of the average week during the busy season. NENA also 
periodically works with providers to study call-traffic patterns so that the industry is aware of any 
material changes and to consider whether the P. 01 standard remains appropriate. 

On January 26, the massive volume of wireless calls to 9-1-1 constituted what the industry 
refers to as a "Focused Overload." A Focused Overload "is generally directed toward a particular 
location and may result from media stimulation (news programs, advertising, call-in contests, 
telethons) or events that cause mass calling to government or public-service agencies, weather 
bureaus, or public utilities." Focused Overloads are not new events. For example, when they 
occur in the network (e.g., from customers dialing in to participate in a TV game show), network 
management controls can be used to limit the volume of calls from connecting to a specific 
destination. Verizon, however, does not employ such controls for 9-1-1 calls. As such, the 
volume of 9-1-1 calls is generally limited by the number of calls that a particular PSAP can answer 
(e.g., if a PSAP has ten 9-1-1 trunks connected to ten PSAP stations handling 9-1-1 calls, it can 
handle ten 9-1-1 calls at a given time). One development in the 9-1-1 sphere is that increasing 
numbers of customers use wireless phones to place calls to 9-1-1 . Frequent redials from the same 
wireless phones, many of which facilitate quick re-dialing, can compound a Focused Overload on 
the trunks carrying wireless 9-1-1 calls to PSAPs. 
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On January 26, Verizon first observed a 9-1-1 PSAP trunk that carries calls from wireless 
carriers removed from service in Montgomery County at 5:45 p.m. When a trunk associated with 
the specific 9-1-1 Selective Routers used by Verizon to serve the counties is taken out of service, 
Verizon's technicians in its Network Operations Center (NOC) receive an alarm. During the 
January 26 storm, Verizon's NOC technicians were inundated with alarms, and these technicians 
worked to restore the trunks that were taken out of service. A number of trunks were restored, 
though many subsequently went back out of service due to the continued Focused Overload. 

As a result of these efforts, while the PSAPs lost trunks that carry calls from wireless 
carriers for certain intervals of time, Verizon's data from the 9-1-1 Selective Router shows that 
some number of wireless calls to the Montgomery County and Prince George's County PSAPs 
were being delivered. By around 1 1 p.m., the number of wireless 9-1-1 calls had abated to the 
extent that all trunks could be restored to working condition. Verizon was able to restore all the 9- 
1-1 wireless trunks without replacing any physical hardware, such as common cards, line cards or 
racks. 

Based on testing, Verizon has concluded that during Focused Overloads to PSAPs, what is 
known as a "double wink" failure caused the PSAP trunks that carry calls from wireless carriers to 
be removed from service during call setup. When a 9-1-1 Selective Router presents a new call to 
the PSAP over an idle trunk, the PSAP equipment (typically a PBX) is configured to respond with 
an "Off-Hook Wink" to indicate that it is able to receive the Automatic Number Identification 
(ANI) digits using Multifrequency (MF) Signaling Tones. If the wink is received within the 
standard sub-second time frame, the call completes. 

Wink failures occur when a Selective Router presents a new call to the PSAP over an idle 
trunk and the PSAP PBX does not respond with a wink to signify its ability to receive the calling 
party's telephone number within the required time frame. If no wink is sent back to the 9-1-1 
Selective Router within the maximum call setup time, it counts one no- wink condition for the 
trunk. During normal operations at the PSAP's average busy hour, if the 9-1-1 Selective Router 
detected a no-wink condition on a trunk, the call would be offered to the next available trunk in a 
round-robin fashion. Since another trunk is likely to be available, that trunk would respond with a 
wink and the call would be completed on the alternative trunk. But, if all the other trunks remain 
busy, as is often the case due to the high volume of calls during a Focused Overload, the same call 
will again be offered to the same trunk where there already has been a "no wink" condition. If this 
second offering results in a second no-wink condition (i.e., a "double wink" failure), the 9-1-1 
Selective Router will remove that trunk from service because it concludes that a call cannot be 
completed on that trunk. When the trunk is taken out of service, an alarm will be presented in 
Verizon's NOC, and a technician will investigate the cause of the trunk failure and attempt to 
restore it. 

With the removal of each trunk, there is a reduction of the call-handling capacity of the 
specific group of trunks serving the PSAP. If the Focused Overload event persists, all trunks 
within the affected trunk group could potentially be taken out of service. 
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The same company manufactures the PBX equipment that Montgomery and Prince 
George's counties use. The age and/or version of such equipment vary, but appear to have had no 
effect on the susceptibility of the PBX to wink failures. Newer equipment, such as Montgomery 
County's PBX, may contain more features, such as being IP-ready, and perform more functions. 
But, the functionality associated with sending winks to the 9-1-1 Selective Router appears to be 
unchanged from older PBXs. Although Verizon has noted wink failures only in instances in which 
traffic is delivered to PBXs from the type of Selective Router used in Montgomery and Prince 
George's counties, Verizon is considering testing another commonly used router, which also 
utilizes winks in its call setup operations, in its lab. 

Finally, Focused Overloads are not unique to the Washington D.C. region. They could 
occur in other geographic areas as well, whether those areas are served by Verizon or another 
provider. In June 2010, Verizon observed what it later determined to be a Focused Overload that 
impacted a PSAP in another state during a severe weather event. Subsequently, Verizon began 
working to understand what occurred and to develop a remediation plan. Verizon has had 
discussions with the PBX and Selective Router vendors, as well as NENA, to further its 
understanding. In August, Verizon began studying the event in Verizon's lab. Verizon has 
recreated a Focused Overload in its lab during which the PBX did not respond in the allotted time 
and has tested several proposed fixes. Verizon observed Focused Overloads in the same state in 
December and February. During the latter event, Verizon was able to test the remediation and 
communications plans described below and to verify that they work. 

Remediation 

After consultations with equipment manufacturers and examining the impact of Focused 
Overloads in Verizon's test lab, Verizon has developed a remediation plan that it is now offering 
all the PSAPs it serves. 

Specifically, Verizon recommends to PSAPs that a Selective Router trunk group setting, 
known as the "automatic busy percentage" setting, be changed to allow only a single trunk in a 
group to be removed from service automatically due to wink failures. When a single trunk is out 
of service, it will signal a critical alarm in Verizon's NOC, which will require a technician to test 
all the trunks to the PSAP to determine if they are working properly. This testing will allow the 
technician to determine if the alarm is related to a Focused Overload and all trunks should remain 
in service. However, if the alarm is not related to a Focused Overload and the trunks fail 
diagnostic tests, the trunks will be taken out of service and calls will be rerouted, if possible. It is 
important to note that even with this remediation plan, during a Focused Overload, a number of 
wireless 9-1-1 calls will still receive busy signals if there are more calls than available trunks. 

Verizon has prepared a Technical Bulletin (Attachment 1) that describes the remediation 
plan and has distributed it to all Maryland PSAPs, as well as the Maryland Emergency Number 
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Systems Board. Verizon will send the bulletin to all the PSAPs it serves and intends to implement 
the plan, as appropriate, with all the PSAPs as soon as practicable. 

Verizon and the equipment manufacturers are continuing work in the lab to examine 
alternatives because certain PSAPs may choose not to implement the remediation plan. For 
example, some PSAPs have automatic alternate routing to call takers when a trunk is removed 
from service for certain reasons unrelated to a Focused Overload, such as from a cable cut. For 
PSAPs that adopt the remediation plan, the rerouting would not take place automatically because 
trunks would not automatically be taken out of service. To address such a situation, a technician in 
the NOC will respond quickly to an alarm and, if appropriate, will manually take trunks out of 
service to reroute the calls. As a result, PSAPs unlikely to experience Focused Overloads, such as 
PSAPs with less dense populations, may want to consult further with Verizon to consider their 
options. 

Finally, to date, Verizon has only observed Focused Overloads causing wink failures of 
CAMA (Centralized Automatic Message Accounting) trunks. CAMA trunking technology has 
had widespread support across many platforms within the industry for many years. CAMA trunks 
were originally designed to provide a method for delivering the calling (originating) number from 
End Office to Toll Office that enabled automatic recording of call time and duration. In the 9-1-1 
context, the numeric digits that reflect the calling number are relayed across the CAMA trunk to 
the PSAP. CAMA trunks utilize in-band signaling for call setup, which causes the Selective 
Router to remove a CAMA trunk from service if call setup is not accomplished (e.g., due to a 
double wink failure). SS7, PRI, and IP-trunking, which rely on out-of-band signaling for call 
setup, do not remove trunks from service for that reason. 

While Verizon will consider the implication of replacing CAMA trunks with IP-based 
trunks/Next Generation 9-1-1 (NG91 1) architecture, it is important to recognize the breadth of 
such a proposal and the substantial impact it would have on PSAPs by requiring them to make 
significant expenditures on equipment, facilities, and training while many are already experiencing 
funding issues. 

Verizon shares the Commission's vision of NG91 1 being available to the entire nation, but 
cautions that much work remains to be done to develop industry standards and to promote the 
widespread adoption of NG91 1 by PSAPs. Verizon is willing to assist in that work as it has long 
been actively engaged in the industry effort to develop standards for NG91 1 . Verizon is also a 
market leader in helping move PSAPs to its IP network. For example, Verizon recently announced 
an effort to combine its global IP network and managed and professional services with Intrado's 
advanced 91 1 technologies to deliver calls and data services to PSAPs via an IP-enabled 
communications platform that will be able to support emerging and future capabilities, such as 
consumers sending photos, videos and text messages. Verizon will provide these services to the 
Denco Area 91 1 District in North Texas in June and intends to offer these services nationwide later 
in 2011. 
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Improved Communication with PSAPs 

On February 1 8, 201 1, Verizon implemented a new procedure to ensure that an affected 
PSAP is notified even more quickly after Verizon becomes aware that even a single 9-1-1 trunk is 
out of service. This notification will occur whether or not there is a Focused Overload event and 
whether or not a particular PSAP is implementing the technical plan explained above. Verizon's 
goal is to notify the PSAP within 1 5 minutes of such an event. Of course, such immediate 
notifications will be preliminary since Verizon may not yet have many details on the nature of the 
particular problem at hand. In such cases, Verizon plans to follow up with the PSAPs to update 
them once the investigation has been completed. 

Similarly, Verizon will notify a PSAP customer when Verizon has an indication that a 
Focused Overload event is occurring. That is, Verizon will continue to manage the network 
trouble that the alarm(s) may indicate, but will not delay getting a notification out to the PSAP 
even before Verizon knows all of the details. This change, coupled with the technological 
adjustments noted, will help keep services functioning during that event, and will provide PSAP 
customers with more timely notice of service issues. Verizon will also prioritize notification for 
other indicated abnormal or reportable alarms related to trunks being out of service. For example, 
in a situation with multiple alarms for the 9-1-1 network, Verizon's NOC will contact the Verizon 
customer care center with the details then available about the alarms (though they will be 
preliminary and subject to further analysis); the customer care center will then be able to 
communicate more quickly to the PSAP. Depending on the scope of the alarms and service 
troubles, Verizon's 9-1-1 customer care center can engage the assistance of additional Verizon 
personnel, such as a 9-1-1 technical service manager, to assist in notifying multiple PSAPs as 
needed. 

Verizon also has implemented another change to its communications processes that will 
help alert the PSAP customer to a Focused Overload event. If the Network Operations Center 
receives alarms that indicate that a wink failure occurs more than five times in any five-minute 
period, and unless another cause is already indicated, Verizon will begin internal diagnostics for a 
possible Focused Overload event. In addition, the NOC will inform Verizon's customer care 
center of the alarm so the care center can notify affected PSAPs. This change will allow for more 
timely Verizon communications with the PSAPs. 

Verizon is continuing to investigate other ways to provide enhanced (and potentially 
automated) notifications to the PSAPs of trunks being out of service. It would require substantial 
development to create a system that allows PSAPs to view NOC data, including alarms when 
trunks are not in service, in real-time. Rather than spending resources retrofitting existing 
installations, it may make more sense to enable such capabilities in the context of migration to 
NG91 1 serving arrangements. In light of Verizon's new procedures to enhance communication 
with the PSAPs and the costs to PSAPs for systems providing real time NOC data, the demand for 
such services is unclear. Today, some PSAPs purchase aftermarket products that provide trunk 
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status data, but that is not tied into Verizon's NOC data. Nonetheless, Verizon will continue to 
examine this issue. 

* * * 

Verizon understands the Commission's interest in this matter. We will continue to work 
cooperatively with the PSAP community to ensure that callers receive the level of service they 
deserve and expect when they call 9-1-1. 

Please contact me if you have any questions. 

Respectfully submitted, 

Attachment 



